{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2-lRt0XOvD5b"
      },
      "source": [
        "To run this, press \"*Runtime*\" and press \"*Run all*\" on a **free** Tesla T4 Google Colab instance!\n",
        "\n",
        "\n",
        "<a href=\"https://github.com/meta-llama/synthetic-data-kit\"><img src=\"https://raw.githubusercontent.com/unslothai/notebooks/refs/heads/main/assets/meta%20round%20logo.png\" width=\"137\"></a>\n",
        "<a href=\"https://unsloth.ai/\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"115\"></a>\n",
        "<a href=\"https://discord.gg/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord button.png\" width=\"145\"></a>\n",
        "<a href=\"https://docs.unsloth.ai/\"><img src=\"https://github.com/unslothai/unsloth/blob/main/images/documentation%20green%20button.png?raw=true\" width=\"125\"></a></a> Join Discord if you need help + ⭐ <i>Star us on <a href=\"https://github.com/unslothai/unsloth\">Github</a> </i> ⭐\n",
        "</div>\n",
        "\n",
        "To install Unsloth on your own computer, follow the installation instructions on our Github page [here](https://docs.unsloth.ai/get-started/installing-+-updating).\n",
        "\n",
        "You will learn how to do [data prep](#Data), how to [train](#Train), how to [run the model](#Inference), & [how to save it](#Save)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Om2qjxs5PSr0"
      },
      "source": [
        "### News"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RCa86oMuPSr0"
      },
      "source": [
        "Unsloth now supports Text-to-Speech (TTS) models. Read our [guide here](https://docs.unsloth.ai/basics/text-to-speech-tts-fine-tuning).\n",
        "\n",
        "Read our **[Gemma 3N Guide](https://docs.unsloth.ai/basics/gemma-3n-how-to-run-and-fine-tune)** and check out our new **[Dynamic 2.0](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs)** quants which outperforms other quantization methods!\n",
        "\n",
        "Visit our docs for all our [model uploads](https://docs.unsloth.ai/get-started/all-our-models) and [notebooks](https://docs.unsloth.ai/get-started/unsloth-notebooks).\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "imwJhjjYPSr0"
      },
      "source": [
        "### Installation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 1,
      "metadata": {
        "id": "uICg0GXWvD5d"
      },
      "outputs": [],
      "source": [
        "%%capture\n",
        "import os\n",
        "if \"COLAB_\" not in \"\".join(os.environ.keys()):\n",
        "    !pip install unsloth vllm\n",
        "    !pip install synthetic-data-kit==0.0.3\n",
        "else:\n",
        "    # [NOTE] Do the below ONLY in Colab! Use [[pip install unsloth vllm]]\n",
        "    !pip install --no-deps unsloth vllm==0.8.5.post1\n",
        "    !pip install \"git+https://github.com/AyushExel/synthetic-data-kit.git@feature/multimodal-lance-vqa\"\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 2,
      "metadata": {
        "id": "4Gf0deTyvD5d"
      },
      "outputs": [],
      "source": [
        "#@title Colab Extra Install { display-mode: \"form\" }\n",
        "%%capture\n",
        "import os\n",
        "if \"COLAB_\" not in \"\".join(os.environ.keys()):\n",
        "    !pip install unsloth vllm\n",
        "else:\n",
        "    !pip install --no-deps unsloth vllm==0.8.5.post1\n",
        "    # [NOTE] Do the below ONLY in Colab! Use [[pip install unsloth vllm]]\n",
        "    # Skip restarting message in Colab\n",
        "    import sys, re, requests; modules = list(sys.modules.keys())\n",
        "    for x in modules: sys.modules.pop(x) if \"PIL\" in x or \"google\" in x else None\n",
        "    !pip install --no-deps bitsandbytes accelerate xformers==0.0.29.post3 peft trl triton cut_cross_entropy unsloth_zoo\n",
        "    !pip install sentencepiece protobuf \"datasets>=3.4.1\" huggingface_hub hf_transfer\n",
        "\n",
        "    # vLLM requirements - vLLM breaks Colab due to reinstalling numpy\n",
        "    f = requests.get(\"https://raw.githubusercontent.com/vllm-project/vllm/refs/heads/main/requirements/common.txt\").content\n",
        "    with open(\"vllm_requirements.txt\", \"wb\") as file:\n",
        "        file.write(re.sub(rb\"(transformers|numpy|xformers)[^\\n]{1,}\\n\", b\"\", f))\n",
        "    !pip install -r vllm_requirements.txt"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UY_WX-T9vD5e"
      },
      "source": [
        "### Unsloth"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iLYE7GQsvD5e"
      },
      "source": [
        "## Primary Goal\n",
        "Our goal is to make Llama 3.2 3B understand the \"Byte Latent Transformer: Patches Scale Better Than Tokens\" [research paper](https://ai.meta.com/research/publications/byte-latent-transformer-patches-scale-better-than-tokens/) that was published in December 2024.\n",
        "\n",
        "We'll use https://github.com/meta-llama/synthetic-data-kit to generate question and answer pairs **fully locally** which will be used for finetuning Llama 3.2 3B!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yV7DyufR51IN"
      },
      "source": [
        "Check if it succeeded:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "C2gQZcr_Wp94",
        "outputId": "86fc7f67-bb35-4af4-dbdc-9d5ecd7143ec"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Loading config from: /Users/ayushchaurasia/Documents/synth-data-kit/synthetic-data-kit/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /Users/ayushchaurasia/Documents/synth-data-kit/synthetic-data-kit/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /Users/ayushchaurasia/Documents/synth-data-kit/synthetic-data-kit/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "\u001b[1;34mEnvironment variable check:\u001b[0m\n",
            "API_ENDPOINT_KEY: Present\n",
            "get_llm_provider returning: api-endpoint\n",
            "API_ENDPOINT_KEY environment variable: Found\n",
            "API key source: Environment variable\n",
            "\u001b[2K\u001b[32m⠏\u001b[0m Checking API endpoint access.....INFO:httpx:HTTP Request: GET https://api.llama.com/v1/models \"HTTP/1.1 200 OK\"\n",
            "\u001b[2K\u001b[32m API endpoint access confirmed\u001b[0m\n",
            "\u001b[2K\u001b[32mUsing custom API base: \u001b[0m\u001b[4;94mhttps://api.llama.com/v1\u001b[0m\n",
            "\u001b[2K\u001b[32mDefault model: Llama-\u001b[0m\u001b[1;36m4\u001b[0m\u001b[32m-Maverick-17B-128E-Instruct-FP8\u001b[0m\n",
            "\u001b[2K\u001b[32m⠏\u001b[0m Checking API endpoint access....\u001b[0m\n",
            "\u001b[1A\u001b[2K"
          ]
        }
      ],
      "source": [
        "!API_ENDPOINT_KEY=\"YOUR_API_KEY\" synthetic-data-kit system-check"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xdl7aPFK55M1"
      },
      "source": [
        "## Document Parsing (PDF, CSV, HTML etc.)\n",
        "Now, let's take the Byte Latent Transformer: Patches Scale Better Than Tokens research paper found at https://arxiv.org/abs/2412.09871 and covert it to Q&A pairs in order to finetune Llama 3.2!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "jKsmav4ERHpj",
        "outputId": "2519db19-2e58-49a9-bde3-4652305269c9"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Collecting playwright\n",
            "  Downloading playwright-1.53.0-py3-none-manylinux1_x86_64.whl.metadata (3.5 kB)\n",
            "Collecting PyPDF2\n",
            "  Downloading pypdf2-3.0.1-py3-none-any.whl.metadata (6.8 kB)\n",
            "Collecting pyee<14,>=13 (from playwright)\n",
            "  Downloading pyee-13.0.0-py3-none-any.whl.metadata (2.9 kB)\n",
            "Requirement already satisfied: greenlet<4.0.0,>=3.1.1 in /usr/local/lib/python3.11/dist-packages (from playwright) (3.2.3)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.11/dist-packages (from pyee<14,>=13->playwright) (4.14.1)\n",
            "Downloading playwright-1.53.0-py3-none-manylinux1_x86_64.whl (45.8 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m45.8/45.8 MB\u001b[0m \u001b[31m14.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pypdf2-3.0.1-py3-none-any.whl (232 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m232.6/232.6 kB\u001b[0m \u001b[31m21.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pyee-13.0.0-py3-none-any.whl (15 kB)\n",
            "Installing collected packages: PyPDF2, pyee, playwright\n",
            "Successfully installed PyPDF2-3.0.1 playwright-1.53.0 pyee-13.0.0\n"
          ]
        }
      ],
      "source": [
        "!pip install playwright PyPDF2"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "6y3eMy3gEQ1b",
        "outputId": "1c3f6cea-dc68-41d1-8711-7b3842e5c890"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Requirement already satisfied: pylance in /usr/local/lib/python3.11/dist-packages (0.31.1)\n",
            "Collecting fitz\n",
            "  Downloading fitz-0.0.1.dev2-py2.py3-none-any.whl.metadata (816 bytes)\n",
            "Requirement already satisfied: pyarrow>=14 in /usr/local/lib/python3.11/dist-packages (from pylance) (18.1.0)\n",
            "Requirement already satisfied: numpy>=1.22 in /usr/local/lib/python3.11/dist-packages (from pylance) (2.0.2)\n",
            "Collecting configobj (from fitz)\n",
            "  Downloading configobj-5.0.9-py2.py3-none-any.whl.metadata (3.2 kB)\n",
            "Collecting configparser (from fitz)\n",
            "  Downloading configparser-7.2.0-py3-none-any.whl.metadata (5.5 kB)\n",
            "Requirement already satisfied: httplib2 in /usr/local/lib/python3.11/dist-packages (from fitz) (0.22.0)\n",
            "Requirement already satisfied: nibabel in /usr/local/lib/python3.11/dist-packages (from fitz) (5.3.2)\n",
            "Collecting nipype (from fitz)\n",
            "  Downloading nipype-1.10.0-py3-none-any.whl.metadata (7.1 kB)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.11/dist-packages (from fitz) (2.2.2)\n",
            "Collecting pyxnat (from fitz)\n",
            "  Downloading pyxnat-1.6.3-py3-none-any.whl.metadata (5.4 kB)\n",
            "Requirement already satisfied: scipy in /usr/local/lib/python3.11/dist-packages (from fitz) (1.15.3)\n",
            "Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /usr/local/lib/python3.11/dist-packages (from httplib2->fitz) (3.2.3)\n",
            "Requirement already satisfied: importlib-resources>=5.12 in /usr/local/lib/python3.11/dist-packages (from nibabel->fitz) (6.5.2)\n",
            "Requirement already satisfied: packaging>=20 in /usr/local/lib/python3.11/dist-packages (from nibabel->fitz) (25.0)\n",
            "Requirement already satisfied: typing-extensions>=4.6 in /usr/local/lib/python3.11/dist-packages (from nibabel->fitz) (4.14.1)\n",
            "Requirement already satisfied: click>=6.6.0 in /usr/local/lib/python3.11/dist-packages (from nipype->fitz) (8.2.1)\n",
            "Requirement already satisfied: networkx>=2.5 in /usr/local/lib/python3.11/dist-packages (from nipype->fitz) (3.5)\n",
            "Collecting prov>=1.5.2 (from nipype->fitz)\n",
            "  Downloading prov-2.1.1-py3-none-any.whl.metadata (3.7 kB)\n",
            "Requirement already satisfied: pydot>=1.2.3 in /usr/local/lib/python3.11/dist-packages (from nipype->fitz) (3.0.4)\n",
            "Requirement already satisfied: python-dateutil>=2.2 in /usr/local/lib/python3.11/dist-packages (from nipype->fitz) (2.9.0.post0)\n",
            "Collecting rdflib>=5.0.0 (from nipype->fitz)\n",
            "  Downloading rdflib-7.1.4-py3-none-any.whl.metadata (11 kB)\n",
            "Requirement already satisfied: simplejson>=3.8.0 in /usr/local/lib/python3.11/dist-packages (from nipype->fitz) (3.20.1)\n",
            "Collecting traits>=6.2 (from nipype->fitz)\n",
            "  Downloading traits-7.0.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.8 kB)\n",
            "Requirement already satisfied: filelock>=3.0.0 in /usr/local/lib/python3.11/dist-packages (from nipype->fitz) (3.18.0)\n",
            "Collecting acres (from nipype->fitz)\n",
            "  Downloading acres-0.5.0-py3-none-any.whl.metadata (6.2 kB)\n",
            "Collecting etelemetry>=0.3.1 (from nipype->fitz)\n",
            "  Downloading etelemetry-0.3.1-py3-none-any.whl.metadata (3.2 kB)\n",
            "Collecting looseversion!=1.2 (from nipype->fitz)\n",
            "  Downloading looseversion-1.3.0-py2.py3-none-any.whl.metadata (4.6 kB)\n",
            "Collecting puremagic (from nipype->fitz)\n",
            "  Downloading puremagic-1.30-py3-none-any.whl.metadata (5.8 kB)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.11/dist-packages (from pandas->fitz) (2025.2)\n",
            "Requirement already satisfied: tzdata>=2022.7 in /usr/local/lib/python3.11/dist-packages (from pandas->fitz) (2025.2)\n",
            "Requirement already satisfied: lxml>=4.3 in /usr/local/lib/python3.11/dist-packages (from pyxnat->fitz) (5.4.0)\n",
            "Requirement already satisfied: requests>=2.20 in /usr/local/lib/python3.11/dist-packages (from pyxnat->fitz) (2.32.3)\n",
            "Requirement already satisfied: pathlib>=1.0 in /usr/local/lib/python3.11/dist-packages (from pyxnat->fitz) (1.0.1)\n",
            "Collecting ci-info>=0.2 (from etelemetry>=0.3.1->nipype->fitz)\n",
            "  Downloading ci_info-0.3.0-py3-none-any.whl.metadata (6.1 kB)\n",
            "Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.11/dist-packages (from python-dateutil>=2.2->nipype->fitz) (1.17.0)\n",
            "Requirement already satisfied: charset-normalizer<4,>=2 in /usr/local/lib/python3.11/dist-packages (from requests>=2.20->pyxnat->fitz) (3.4.2)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.11/dist-packages (from requests>=2.20->pyxnat->fitz) (3.10)\n",
            "Requirement already satisfied: urllib3<3,>=1.21.1 in /usr/local/lib/python3.11/dist-packages (from requests>=2.20->pyxnat->fitz) (2.4.0)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.11/dist-packages (from requests>=2.20->pyxnat->fitz) (2025.7.14)\n",
            "Downloading fitz-0.0.1.dev2-py2.py3-none-any.whl (20 kB)\n",
            "Downloading configobj-5.0.9-py2.py3-none-any.whl (35 kB)\n",
            "Downloading configparser-7.2.0-py3-none-any.whl (17 kB)\n",
            "Downloading nipype-1.10.0-py3-none-any.whl (3.2 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m3.2/3.2 MB\u001b[0m \u001b[31m82.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading pyxnat-1.6.3-py3-none-any.whl (95 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m95.4/95.4 kB\u001b[0m \u001b[31m9.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading etelemetry-0.3.1-py3-none-any.whl (6.4 kB)\n",
            "Downloading looseversion-1.3.0-py2.py3-none-any.whl (8.2 kB)\n",
            "Downloading prov-2.1.1-py3-none-any.whl (425 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m425.9/425.9 kB\u001b[0m \u001b[31m33.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading rdflib-7.1.4-py3-none-any.whl (565 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m565.1/565.1 kB\u001b[0m \u001b[31m42.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading traits-7.0.2-cp311-cp311-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (5.1 MB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m5.1/5.1 MB\u001b[0m \u001b[31m58.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading acres-0.5.0-py3-none-any.whl (12 kB)\n",
            "Downloading puremagic-1.30-py3-none-any.whl (43 kB)\n",
            "\u001b[2K   \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m43.3/43.3 kB\u001b[0m \u001b[31m4.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25hDownloading ci_info-0.3.0-py3-none-any.whl (7.8 kB)\n",
            "Installing collected packages: puremagic, looseversion, traits, rdflib, configparser, configobj, ci-info, acres, pyxnat, prov, etelemetry, nipype, fitz\n",
            "Successfully installed acres-0.5.0 ci-info-0.3.0 configobj-5.0.9 configparser-7.2.0 etelemetry-0.3.1 fitz-0.0.1.dev2 looseversion-1.3.0 nipype-1.10.0 prov-2.1.1 puremagic-1.30 pyxnat-1.6.3 rdflib-7.1.4 traits-7.0.2\n"
          ]
        },
        {
          "data": {
            "application/vnd.colab-display-data+json": {
              "id": "ce520d84af6544de9d9d2f45e990240e",
              "pip_warning": {
                "packages": [
                  "backports"
                ]
              }
            }
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "!pip install pylance fitz"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "8BN1yrPGmANA",
        "outputId": "3c85889d-4ffb-4b30-e0a5-c9515341d22e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Loading config from: /Users/ayushchaurasia/Documents/synth-data-kit/synthetic-data-kit/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /Users/ayushchaurasia/Documents/synth-data-kit/synthetic-data-kit/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /Users/ayushchaurasia/Documents/synth-data-kit/synthetic-data-kit/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "\u001b[2K\u001b[32m⠙\u001b[0m Processing rl-guide.pdf...\u001b[90m[\u001b[0m2025-07-18T19:49:41Z \u001b[33mWARN \u001b[0m lance::dataset::write::insert\u001b[90m]\u001b[0m No existing dataset at data/parsed/lora.lance, it will be created\n",
            "\u001b[2K\u001b[32m⠹\u001b[0m Processing rl-guide.pdf...\n",
            "\u001b[1A\u001b[2K\u001b[32m✅ Text successfully extracted to \u001b[0m\u001b[1;32mdata/parsed/lora.lance\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "# Byte Latent Transformer: Patches Scale Better Than Tokens paper in HTML format\n",
        "!synthetic-data-kit \\\n",
        "    ingest  \"rl-gu--multimodalide.pdf\" \\\n",
        "    --name \"lora\"\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CnDmnEzL5wMW"
      },
      "source": [
        "### Peek at the parsed multimodal data in lance format"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "id": "E4GMrZly5431"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "24\n"
          ]
        },
        {
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>text</th>\n",
              "      <th>image</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1\\nText-to-speech, all model types &amp; full fine...</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2\\nThat's it! There are intricacies on what \"g...</td>\n",
              "      <td>b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3\\nAnother example is imagine you are given th...</td>\n",
              "      <td>b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4\\nOpenAI popularized the concept of RLHF\\n (R...</td>\n",
              "      <td>b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4\\nOpenAI popularized the concept of RLHF\\n (R...</td>\n",
              "      <td>b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>4\\nOpenAI popularized the concept of RLHF\\n (R...</td>\n",
              "      <td>b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>5\\nThe clip(..., 1-e, 1+e) term is used to for...</td>\n",
              "      <td>b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>6\\nThis means GRPO is extremely efficient. Pre...</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8</th>\n",
              "      <td>7\\nFor example for \"What is 2+2?\" we sample 4 ...</td>\n",
              "      <td>b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9</th>\n",
              "      <td>7\\nFor example for \"What is 2+2?\" we sample 4 ...</td>\n",
              "      <td>b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>10</th>\n",
              "      <td>8\\nThe trick of RL is you need 2 things only:\\...</td>\n",
              "      <td>b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11</th>\n",
              "      <td>9\\nIn the \"What is 2+2?\" example - 0, cat, -10...</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>12</th>\n",
              "      <td>10\\nNEW! We now support Dr. GRPO and most othe...</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>13</th>\n",
              "      <td>11\\nFor a tutorial on how to transform any ope...</td>\n",
              "      <td>b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>14</th>\n",
              "      <td>12\\n• Wait for at least 300 steps for the rewa...</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>15</th>\n",
              "      <td>13\\n• Training loss tracking for GRPO is now b...</td>\n",
              "      <td>b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16</th>\n",
              "      <td>13\\n• Training loss tracking for GRPO is now b...</td>\n",
              "      <td>b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>14\\nReward Function:\\n• Converts verification ...</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>15\\nYou can refer to the examples below. You c...</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>16\\nIf you’ve checked out our Advanced GRPO Co...</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>20</th>\n",
              "      <td>17\\nYou can now use vLLM\\n directly in your fi...</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>21</th>\n",
              "      <td>18\\n1. For GRPO's GPU VRAM requirements for QL...</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>22</th>\n",
              "      <td>19\\nset the batch size for vLLM to 8, but we s...</td>\n",
              "      <td>None</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>23</th>\n",
              "      <td>20\\nCommunity\\nReddit r/unsloth\\nTwitter (X)\\n...</td>\n",
              "      <td>b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                                 text  \\\n",
              "0   1\\nText-to-speech, all model types & full fine...   \n",
              "1   2\\nThat's it! There are intricacies on what \"g...   \n",
              "2   3\\nAnother example is imagine you are given th...   \n",
              "3   4\\nOpenAI popularized the concept of RLHF\\n (R...   \n",
              "4   4\\nOpenAI popularized the concept of RLHF\\n (R...   \n",
              "5   4\\nOpenAI popularized the concept of RLHF\\n (R...   \n",
              "6   5\\nThe clip(..., 1-e, 1+e) term is used to for...   \n",
              "7   6\\nThis means GRPO is extremely efficient. Pre...   \n",
              "8   7\\nFor example for \"What is 2+2?\" we sample 4 ...   \n",
              "9   7\\nFor example for \"What is 2+2?\" we sample 4 ...   \n",
              "10  8\\nThe trick of RL is you need 2 things only:\\...   \n",
              "11  9\\nIn the \"What is 2+2?\" example - 0, cat, -10...   \n",
              "12  10\\nNEW! We now support Dr. GRPO and most othe...   \n",
              "13  11\\nFor a tutorial on how to transform any ope...   \n",
              "14  12\\n• Wait for at least 300 steps for the rewa...   \n",
              "15  13\\n• Training loss tracking for GRPO is now b...   \n",
              "16  13\\n• Training loss tracking for GRPO is now b...   \n",
              "17  14\\nReward Function:\\n• Converts verification ...   \n",
              "18  15\\nYou can refer to the examples below. You c...   \n",
              "19  16\\nIf you’ve checked out our Advanced GRPO Co...   \n",
              "20  17\\nYou can now use vLLM\\n directly in your fi...   \n",
              "21  18\\n1. For GRPO's GPU VRAM requirements for QL...   \n",
              "22  19\\nset the batch size for vLLM to 8, but we s...   \n",
              "23  20\\nCommunity\\nReddit r/unsloth\\nTwitter (X)\\n...   \n",
              "\n",
              "                                                image  \n",
              "0                                                None  \n",
              "1   b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...  \n",
              "2   b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...  \n",
              "3   b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...  \n",
              "4   b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...  \n",
              "5   b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...  \n",
              "6   b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...  \n",
              "7                                                None  \n",
              "8   b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...  \n",
              "9   b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...  \n",
              "10  b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...  \n",
              "11                                               None  \n",
              "12                                               None  \n",
              "13  b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...  \n",
              "14                                               None  \n",
              "15  b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...  \n",
              "16  b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...  \n",
              "17                                               None  \n",
              "18                                               None  \n",
              "19                                               None  \n",
              "20                                               None  \n",
              "21                                               None  \n",
              "22                                               None  \n",
              "23  b'\\x89PNG\\r\\n\\x1a\\n\\x00\\x00\\x00\\rIHDR\\x00\\x00\\...  "
            ]
          },
          "execution_count": 3,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "import lance\n",
        "\n",
        "ds = lance.dataset(\"data/parsed/lora.lance\")\n",
        "print(ds.count_rows())\n",
        "ds.to_table().to_pandas()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nGdAXafV6S2M"
      },
      "source": [
        "We see around 37 chunks of data. We now call synthetic-data-kit to create some pairs of data for 3 of our chunks.\n",
        "\n",
        "You can process more chunks, but it'll be much slower!\n",
        "\n",
        "Using `--num-pairs` will generate **approximately** that many QA pairs. However it might be shorter or longer depending on the `max_seq_length` of the loaded up model. So if you specify 100, you might only get 10 since the model's max sequence length is capped."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "pYYYlMJ7ZtT7",
        "outputId": "d2c8a485-5f6f-41a7-b296-8e9a5be3a113"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Loading config from: /Users/ayushchaurasia/Documents/synth-data-kit/synthetic-data-kit/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /Users/ayushchaurasia/Documents/synth-data-kit/synthetic-data-kit/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /Users/ayushchaurasia/Documents/synth-data-kit/synthetic-data-kit/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "get_llm_provider returning: api-endpoint\n",
            "\u001b[32m🔗 Using api-endpoint provider\u001b[0m\n",
            "\u001b[?25lLoading config from: \n",
            "/Users/ayushchaurasia/Documents/synth-data-kit/synthetic-data-kit/synthetic_data\n",
            "_kit/config.yaml\n",
            "\u001b[2KConfig has LLM provider set to: api-endpointed/lora.lance...\n",
            "\u001b[2KAPI_ENDPOINT_KEY from environment: Foundparsed/lora.lance...\n",
            "\u001b[2KUsing API key: From env varnt from data/parsed/lora.lance...\n",
            "\u001b[2KUsing API base URL: https://api.llama.com/v1ed/lora.lance...\n",
            "\u001b[2KL Using api-endpoint provider from data/parsed/lora.lance...\n",
            "\u001b[2KLoading config from:  content from data/parsed/lora.lance...\n",
            "/Users/ayushchaurasia/Documents/synth-data-kit/synthetic-data-kit/synthetic_data\n",
            "_kit/config.yaml\n",
            "\u001b[2KConfig has LLM provider set to: api-endpointed/lora.lance...\n",
            "\u001b[2KParameter 'function'=<bound method VQAGenerator.transform of ...\n",
            "<synthetic_data_kit.generators.vqa_generator.VQAGenerator object at \n",
            "0x109673df0>> of the transform datasets.arrow_dataset.Dataset._map_single \n",
            "couldn't be hashed properly, a random hash was used instead. Make sure your \n",
            "transforms and parameters are serializable with pickle or dill for the dataset \n",
            "fingerprinting and caching to work. If you reuse this transform, the caching \n",
            "mechanism will consider it to be different from the previous calls and recompute\n",
            "everything. This warning is only showed once. Subsequent hashing failures won't \n",
            "be showed.\n",
            "\u001b[32m⠼\u001b[0m Generating vqa content from data/parsed/lora.lance...WARNING:datasets.fingerprint:Parameter 'function'=<bound method VQAGenerator.transform of <synthetic_data_kit.generators.vqa_generator.VQAGenerator object at 0x109673df0>> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\n",
            "\u001b[2KMap:   \u001b[1;36m0\u001b[0m%|                                        | \u001b[1;36m0\u001b[0m/\u001b[1;36m24\u001b[0m \u001b[1m[\u001b[0m\u001b[1;92m00:00\u001b[0m<?, ? examples/s\u001b[1m]\u001b[0m\n",
            "\u001b[2KMap:   \u001b[1;36m0\u001b[0m%|                                        | \u001b[1;36m0\u001b[0m/\u001b[1;36m24\u001b[0m \u001b[1m[\u001b[0m\u001b[1;92m00:00\u001b[0m<?, ? examples/s\u001b[1m]\u001b[0m\n",
            "\u001b[2Km⠼\u001b[0m Generating vqa content from data/parsed/lora.lance...\n",
            "\u001b[2K\u001b[32m⠼\u001b[0m Generating vqa content from data/parsed/lora.lance...\n",
            "\u001b[1A\u001b[2K\u001b[31m❌ Error: \u001b[0m\u001b[32m'query'\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "\n",
        "!API_ENDPOINT_KEY=\"YOUR_API_KEY\" synthetic-data-kit create \"data/parsed/lora.lance\" --num-pairs 25 --type \"multimodal-qa\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "cNkxxvBx7Csp"
      },
      "source": [
        "Optionally, you can clean up the data via pruning \"bad\" or low quality examples and convert the rest to JSON format for finetuning!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "HMD-izj5OiAK",
        "outputId": "fedb3693-6325-4909-d973-16cf90f985d1"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "get_llm_provider returning: api-endpoint\n",
            "\u001b[32m🔗 Using api-endpoint provider\u001b[0m\n",
            "\u001b[?25l\u001b[32m⠋\u001b[0m Cleaning content from data/generated/reasoning_qa_multimodal.json...\n",
            "\u001b[?25h\r\u001b[1A\u001b[2K\u001b[31m❌ Error: \u001b[0m\u001b[1;31m[\u001b[0m\u001b[31mErrno \u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;31m]\u001b[0m\u001b[31m No such file or directory: \u001b[0m\n",
            "\u001b[32m'data/generated/reasoning_qa_multimodal.json'\u001b[0m\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "get_llm_provider returning: api-endpoint\n",
            "\u001b[32m🔗 Using api-endpoint provider\u001b[0m\n",
            "\u001b[?25l\u001b[32m⠋\u001b[0m Cleaning content from data/generated/lora_qa_multimodal.json...\n",
            "\u001b[1A\u001b[2K\u001b[31m❌ Error: \u001b[0m\u001b[1;31m[\u001b[0m\u001b[31mErrno \u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;31m]\u001b[0m\u001b[31m No such file or directory: \u001b[0m\n",
            "\u001b[32m'data/generated/lora_qa_multimodal.json'\u001b[0m\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "get_llm_provider returning: api-endpoint\n",
            "\u001b[32m🔗 Using api-endpoint provider\u001b[0m\n",
            "\u001b[?25l\u001b[32m⠋\u001b[0m Cleaning content from data/generated/deepseek-local_qa_multimodal.json...\n",
            "\u001b[1A\u001b[2K\u001b[31m❌ Error: \u001b[0m\u001b[1;31m[\u001b[0m\u001b[31mErrno \u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;31m]\u001b[0m\u001b[31m No such file or directory: \u001b[0m\n",
            "\u001b[32m'data/generated/deepseek-local_qa_multimodal.json'\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "for filename in [\n",
        "                  \"data/generated/reasoning_qa_multimodal.json\",\n",
        "                  \"data/generated/lora_qa_multimodal.json\",\n",
        "                  \"data/generated/deepseek-local_qa_multimodal.json\"\n",
        "                  ]:\n",
        "  !API_ENDPOINT_KEY=\"YOUR_API_KEY\" synthetic-data-kit \\\n",
        "      curate --threshold 5.0 {filename}\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AScJ5-vAOjYj"
      },
      "source": [
        "We now convert the generated datasets into QA formats so we can load it for finetuning:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "J9Um4Z8SqUTB",
        "outputId": "9bd14e88-aa3f-4cff-c8cc-5d3f5223c062"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "INFO:datasets:PyTorch version 2.6.0+cu124 available.\n",
            "INFO:datasets:Polars version 1.21.0 available.\n",
            "INFO:datasets:Duckdb version 1.2.2 available.\n",
            "INFO:datasets:TensorFlow version 2.18.0 available.\n",
            "INFO:datasets:JAX version 0.5.2 available.\n",
            "\u001b[?25l\u001b[32m⠋\u001b[0m Converting data/cleaned/deepseek-local_qa_multimodal_cleaned.json to ft format\n",
            "with lance storage...\n",
            "\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[31m❌ Error: \u001b[0m\u001b[1;31m[\u001b[0m\u001b[31mErrno \u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;31m]\u001b[0m\u001b[31m No such file or directory: \u001b[0m\n",
            "\u001b[32m'data/cleaned/deepseek-local_qa_multimodal_cleaned.json'\u001b[0m\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "INFO:datasets:PyTorch version 2.6.0+cu124 available.\n",
            "INFO:datasets:Polars version 1.21.0 available.\n",
            "INFO:datasets:Duckdb version 1.2.2 available.\n",
            "INFO:datasets:TensorFlow version 2.18.0 available.\n",
            "INFO:datasets:JAX version 0.5.2 available.\n",
            "\u001b[?25l\u001b[32m⠋\u001b[0m Converting data/cleaned/lora_qa_multimodal_cleaned.json to ft format with \n",
            "lance storage...\n",
            "\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[31m❌ Error: \u001b[0m\u001b[1;31m[\u001b[0m\u001b[31mErrno \u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;31m]\u001b[0m\u001b[31m No such file or directory: \u001b[0m\n",
            "\u001b[32m'data/cleaned/lora_qa_multimodal_cleaned.json'\u001b[0m\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "Loading config from: /usr/local/lib/python3.11/dist-packages/synthetic_data_kit/config.yaml\n",
            "Config has LLM provider set to: api-endpoint\n",
            "INFO:datasets:PyTorch version 2.6.0+cu124 available.\n",
            "INFO:datasets:Polars version 1.21.0 available.\n",
            "INFO:datasets:Duckdb version 1.2.2 available.\n",
            "INFO:datasets:TensorFlow version 2.18.0 available.\n",
            "INFO:datasets:JAX version 0.5.2 available.\n",
            "\u001b[?25l\u001b[32m⠋\u001b[0m Converting data/cleaned/reasoning_qa_multimodal_cleaned.json to ft format with\n",
            "lance storage...\n",
            "\u001b[1A\u001b[2K\u001b[1A\u001b[2K\u001b[31m❌ Error: \u001b[0m\u001b[1;31m[\u001b[0m\u001b[31mErrno \u001b[0m\u001b[1;36m2\u001b[0m\u001b[1;31m]\u001b[0m\u001b[31m No such file or directory: \u001b[0m\n",
            "\u001b[32m'data/cleaned/reasoning_qa_multimodal_cleaned.json'\u001b[0m\n"
          ]
        }
      ],
      "source": [
        "for filename in [\n",
        "    \"data/cleaned/deepseek-local_qa_multimodal_cleaned.json\",\n",
        "    \"data/cleaned/lora_qa_multimodal_cleaned.json\",\n",
        "    \"data/cleaned/reasoning_qa_multimodal_cleaned.json\"]:\n",
        "  !synthetic-data-kit \\\n",
        "        save-as {filename} -f ft --storage lance"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6dVK-qza7rPB"
      },
      "source": [
        "Let's load up the data and see what the synthetic data looks like!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 408
        },
        "id": "VrBwG2KT7dam",
        "outputId": "e703edd7-a907-4b1e-b3bf-4b723ba271df"
      },
      "outputs": [
        {
          "ename": "ValueError",
          "evalue": "Dataset at path content/data/final/deepseek-local_qa_multimodal_cleaned_ft.lance was not found: Not found: content/data/final/deepseek-local_qa_multimodal_cleaned_ft.lance/_versions, /home/runner/work/lance/lance/rust/lance-table/src/io/commit.rs:328:23, /home/runner/work/lance/lance/rust/lance/src/dataset/builder.rs:329:35",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
            "\u001b[0;32m/tmp/ipython-input-10-2209798120.py\u001b[0m in \u001b[0;36m<cell line: 0>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      8\u001b[0m ]\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 10\u001b[0;31m conversations = pd.concat([\n\u001b[0m\u001b[1;32m     11\u001b[0m     \u001b[0mlance\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_table\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_pandas\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mfilename\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mfinal_filenames\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     12\u001b[0m ]).reset_index(drop = True)\n",
            "\u001b[0;32m/tmp/ipython-input-10-2209798120.py\u001b[0m in \u001b[0;36m<listcomp>\u001b[0;34m(.0)\u001b[0m\n\u001b[1;32m      9\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m conversations = pd.concat([\n\u001b[0;32m---> 11\u001b[0;31m     \u001b[0mlance\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdataset\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfilename\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_table\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mto_pandas\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mfilename\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mfinal_filenames\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m ]).reset_index(drop = True)\n\u001b[1;32m     13\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/lance/__init__.py\u001b[0m in \u001b[0;36mdataset\u001b[0;34m(uri, version, asof, block_size, commit_lock, index_cache_size, storage_options, default_scan_options, metadata_cache_size_bytes)\u001b[0m\n\u001b[1;32m    129\u001b[0m         \u001b[0ma\u001b[0m \u001b[0mdefault\u001b[0m \u001b[0msize\u001b[0m \u001b[0mwill\u001b[0m \u001b[0mbe\u001b[0m \u001b[0mused\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    130\u001b[0m     \"\"\"\n\u001b[0;32m--> 131\u001b[0;31m     ds = LanceDataset(\n\u001b[0m\u001b[1;32m    132\u001b[0m         \u001b[0muri\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    133\u001b[0m         \u001b[0mversion\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.11/dist-packages/lance/dataset.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, uri, version, block_size, index_cache_size, metadata_cache_size, commit_lock, storage_options, serialized_manifest, default_scan_options, metadata_cache_size_bytes)\u001b[0m\n\u001b[1;32m    226\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_uri\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0muri\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    227\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_storage_options\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstorage_options\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 228\u001b[0;31m         self._ds = _Dataset(\n\u001b[0m\u001b[1;32m    229\u001b[0m             \u001b[0muri\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    230\u001b[0m             \u001b[0mversion\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mValueError\u001b[0m: Dataset at path content/data/final/deepseek-local_qa_multimodal_cleaned_ft.lance was not found: Not found: content/data/final/deepseek-local_qa_multimodal_cleaned_ft.lance/_versions, /home/runner/work/lance/lance/rust/lance-table/src/io/commit.rs:328:23, /home/runner/work/lance/lance/rust/lance/src/dataset/builder.rs:329:35"
          ]
        }
      ],
      "source": [
        "from datasets import Dataset\n",
        "import pandas as pd\n",
        "import lance\n",
        "final_filenames = [\n",
        " \"data/final/deepseek-local_qa_multimodal_cleaned_ft.lance\",\n",
        " \"data/final/lora_qa_multimodal_cleaned_ft.lance\",\n",
        " \"data/final/reasoning_qa_multimodal_cleaned_ft.lance\"\n",
        "]\n",
        "\n",
        "conversations = pd.concat([\n",
        "    lance.dataset(filename).to_table().to_pandas() for filename in final_filenames\n",
        "]).reset_index(drop = True)\n",
        "\n",
        "dataset = Dataset.from_pandas(conversations)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "jaZ3tRP8frSn"
      },
      "outputs": [],
      "source": [
        "dataset[11]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "504n46Sxfruu"
      },
      "outputs": [],
      "source": [
        "dataset[20]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1BVBp9YXRw_o"
      },
      "outputs": [],
      "source": [
        "dataset[300]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aO9qePmP7yaY"
      },
      "source": [
        "Finally free vLLM process to save memory and to allow for finetuning!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EQo2PR7oqDQE"
      },
      "source": [
        "### Fine-tuning Synthetic Dataset with Unsloth"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "QmUBVEnvCDJv"
      },
      "outputs": [],
      "source": [
        "from unsloth import FastLanguageModel\n",
        "import torch\n",
        "\n",
        "fourbit_models = [\n",
        "    # 4bit dynamic quants for superior accuracy and low memory use\n",
        "    \"unsloth/gemma-3-4b-it-unsloth-bnb-4bit\",\n",
        "    \"unsloth/gemma-3-12b-it-unsloth-bnb-4bit\",\n",
        "    \"unsloth/gemma-3-27b-it-unsloth-bnb-4bit\",\n",
        "    # Qwen3 new models\n",
        "    \"unsloth/Qwen3-4B-unsloth-bnb-4bit\",\n",
        "    \"unsloth/Qwen3-8B-unsloth-bnb-4bit\",\n",
        "    # Other very popular models!\n",
        "    \"unsloth/Llama-3.1-8B\",\n",
        "    \"unsloth/Llama-3.2-3B\",\n",
        "    \"unsloth/Llama-3.3-70B\",\n",
        "    \"unsloth/mistral-7b-instruct-v0.3\",\n",
        "    \"unsloth/Phi-4\",\n",
        "] # More models at https://huggingface.co/unsloth\n",
        "\n",
        "model, tokenizer = FastLanguageModel.from_pretrained(\n",
        "    model_name = \"unsloth/Llama-3.2-3B-Instruct\",\n",
        "    max_seq_length = 2048, # Choose any for long context!\n",
        "    load_in_4bit = True,  # 4 bit quantization to reduce memory\n",
        "    load_in_8bit = False, # [NEW!] A bit more accurate, uses 2x memory\n",
        "    full_finetuning = False, # [NEW!] We have full finetuning now!\n",
        "    # token = \"hf_...\", # use one if using gated models\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SXd9bTZd1aaL"
      },
      "source": [
        "We now add LoRA adapters so we only need to update 1 to 10% of all parameters!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6bZsfBuZDeCL"
      },
      "outputs": [],
      "source": [
        "model = FastLanguageModel.get_peft_model(\n",
        "    model,\n",
        "    r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128\n",
        "    target_modules = [\"q_proj\", \"k_proj\", \"v_proj\", \"o_proj\",\n",
        "                      \"gate_proj\", \"up_proj\", \"down_proj\",],\n",
        "    lora_alpha = 16,\n",
        "    lora_dropout = 0, # Supports any, but = 0 is optimized\n",
        "    bias = \"none\",    # Supports any, but = \"none\" is optimized\n",
        "    # [NEW] \"unsloth\" uses 30% less VRAM, fits 2x larger batch sizes!\n",
        "    use_gradient_checkpointing = \"unsloth\", # True or \"unsloth\" for very long context\n",
        "    random_state = 3407,\n",
        "    use_rslora = False,  # We support rank stabilized LoRA\n",
        "    loftq_config = None, # And LoftQ\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vITh0KVJ10qX"
      },
      "source": [
        "<a name=\"Data\"></a>\n",
        "### Data Prep\n",
        "We now use the `Llama-3.2` format for conversation style finetunes. The chat template renders conversations like below: (Cutting Knowledge Date is by default there!)\n",
        "\n",
        "```\n",
        "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n",
        "\n",
        "Cutting Knowledge Date: December 2023\n",
        "Today Date: 01 May 2025\n",
        "\n",
        "You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>\n",
        "\n",
        "What is 1+1?<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n",
        "\n",
        "2<|eot_id|>\n",
        "```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "L0uAoIgJ203D"
      },
      "outputs": [],
      "source": [
        "dataset[0]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LjY75GoYUCB8"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "def formatting_prompts_func(examples):\n",
        "    # Parse the JSON string in 'messages' into a Python list of dictionaries\n",
        "    convos = [json.loads(message_str) for message_str in examples[\"messages\"]]\n",
        "    texts = [tokenizer.apply_chat_template(convo, tokenize=False, add_generation_prompt=False) for convo in convos]\n",
        "    return { \"text\" : texts, }\n",
        "\n",
        "# Get our previous dataset and format it:\n",
        "dataset = dataset.map(formatting_prompts_func, batched = True,)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YKA0VEF4CfCB"
      },
      "source": [
        "See result of the first row:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0usAI0M40hpT"
      },
      "outputs": [],
      "source": [
        "dataset[0]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "idAEIeSQ3xdS"
      },
      "source": [
        "<a name=\"Train\"></a>\n",
        "### Train the model\n",
        "Now let's use Huggingface TRL's `SFTTrainer`! More docs here: [TRL SFT docs](https://huggingface.co/docs/trl/sft_trainer). We do 60 steps to speed things up, but you can set `num_train_epochs=1` for a full run, and turn off `max_steps=None`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "95_Nn-89DhsL"
      },
      "outputs": [],
      "source": [
        "from trl import SFTTrainer, SFTConfig\n",
        "\n",
        "trainer = SFTTrainer(\n",
        "    model = model,\n",
        "    tokenizer = tokenizer,\n",
        "    train_dataset = dataset,\n",
        "    eval_dataset = None, # Can set up evaluation!\n",
        "    args = SFTConfig(\n",
        "        dataset_text_field = \"text\",\n",
        "        per_device_train_batch_size = 2,\n",
        "        gradient_accumulation_steps = 4, # Use GA to mimic batch size!\n",
        "        warmup_steps = 5,\n",
        "        max_steps = 60,\n",
        "        learning_rate = 2e-4,\n",
        "        logging_steps = 1,\n",
        "        optim = \"adamw_8bit\",\n",
        "        weight_decay = 0.01,\n",
        "        lr_scheduler_type = \"linear\",\n",
        "        seed = 3407,\n",
        "        report_to = \"none\", # Use this for WandB etc\n",
        "    ),\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "2ejIt2xSNKKp"
      },
      "outputs": [],
      "source": [
        "# @title Show current memory stats\n",
        "gpu_stats = torch.cuda.get_device_properties(0)\n",
        "start_gpu_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
        "max_memory = round(gpu_stats.total_memory / 1024 / 1024 / 1024, 3)\n",
        "print(f\"GPU = {gpu_stats.name}. Max memory = {max_memory} GB.\")\n",
        "print(f\"{start_gpu_memory} GB of memory reserved.\")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "yqxqAZ7KJ4oL"
      },
      "outputs": [],
      "source": [
        "trainer_stats = trainer.train()"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "pCqnaKmlO1U9"
      },
      "outputs": [],
      "source": [
        "# @title Show final memory and time stats\n",
        "used_memory = round(torch.cuda.max_memory_reserved() / 1024 / 1024 / 1024, 3)\n",
        "used_memory_for_lora = round(used_memory - start_gpu_memory, 3)\n",
        "used_percentage = round(used_memory / max_memory * 100, 3)\n",
        "lora_percentage = round(used_memory_for_lora / max_memory * 100, 3)\n",
        "print(f\"{trainer_stats.metrics['train_runtime']} seconds used for training.\")\n",
        "print(\n",
        "    f\"{round(trainer_stats.metrics['train_runtime']/60, 2)} minutes used for training.\"\n",
        ")\n",
        "print(f\"Peak reserved memory = {used_memory} GB.\")\n",
        "print(f\"Peak reserved memory for training = {used_memory_for_lora} GB.\")\n",
        "print(f\"Peak reserved memory % of max memory = {used_percentage} %.\")\n",
        "print(f\"Peak reserved memory for training % of max memory = {lora_percentage} %.\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ekOmTR1hSNcr"
      },
      "source": [
        "<a name=\"Inference\"></a>\n",
        "### Inference\n",
        "Let's run the model! Use `apply_chat_template` with `add_generation_prompt` set to `True` for inference."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kR3gIAX-SM2q"
      },
      "outputs": [],
      "source": [
        "messages = [\n",
        "    {\"role\": \"user\", \"content\": \"What model types are supported for text-to-speech?\"},\n",
        "]\n",
        "inputs = tokenizer.apply_chat_template(\n",
        "    messages,\n",
        "    tokenize = True,\n",
        "    add_generation_prompt = True, # Must add for generation\n",
        "    return_tensors = \"pt\",\n",
        ").to(\"cuda\")\n",
        "\n",
        "from transformers import TextStreamer\n",
        "text_streamer = TextStreamer(tokenizer, skip_prompt = True)\n",
        "_ = model.generate(input_ids = inputs, streamer = text_streamer,\n",
        "                   max_new_tokens = 256, temperature = 0.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bRrr--20Udm9"
      },
      "source": [
        "The model learns about the research paper!!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2strt31SUc5W"
      },
      "outputs": [],
      "source": [
        "messages = [\n",
        "    {\"role\": \"user\", \"content\": \"What is the BitsType for IQ1_S??\"},\n",
        "]\n",
        "inputs = tokenizer.apply_chat_template(\n",
        "    messages,\n",
        "    tokenize = True,\n",
        "    add_generation_prompt = True, # Must add for generation\n",
        "    return_tensors = \"pt\",\n",
        ").to(\"cuda\")\n",
        "\n",
        "from transformers import TextStreamer\n",
        "text_streamer = TextStreamer(tokenizer, skip_prompt = True)\n",
        "_ = model.generate(input_ids = inputs, streamer = text_streamer,\n",
        "                   max_new_tokens = 256, temperature = 0.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uMuVrWbjAzhc"
      },
      "source": [
        "<a name=\"Save\"></a>\n",
        "### Saving, loading finetuned models\n",
        "To save the final model as LoRA adapters, either use Huggingface's `push_to_hub` for an online save or `save_pretrained` for a local save.\n",
        "\n",
        "**[NOTE]** This ONLY saves the LoRA adapters, and not the full model. To save to 16bit or GGUF, scroll down!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "upcOlWe7A1vc"
      },
      "outputs": [],
      "source": [
        "model.save_pretrained(\"lora_model\")  # Local saving\n",
        "tokenizer.save_pretrained(\"lora_model\")\n",
        "# model.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving\n",
        "# tokenizer.push_to_hub(\"your_name/lora_model\", token = \"...\") # Online saving"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AEEcJ4qfC7Lp"
      },
      "source": [
        "Now if you want to load the LoRA adapters we just saved for inference, set `False` to `True`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "MKX_XKs_BNZR"
      },
      "outputs": [],
      "source": [
        "if False:\n",
        "    from unsloth import FastLanguageModel\n",
        "    model, tokenizer = FastLanguageModel.from_pretrained(\n",
        "        model_name = \"lora_model\", # YOUR MODEL YOU USED FOR TRAINING\n",
        "        max_seq_length = max_seq_length,\n",
        "        dtype = dtype,\n",
        "        load_in_4bit = load_in_4bit,\n",
        "    )\n",
        "messages = [\n",
        "    {\"role\": \"user\", \"content\": \"What is so special about BLT's tokenization?\"},\n",
        "]\n",
        "inputs = tokenizer.apply_chat_template(\n",
        "    messages,\n",
        "    tokenize = True,\n",
        "    add_generation_prompt = True, # Must add for generation\n",
        "    return_tensors = \"pt\",\n",
        ").to(\"cuda\")\n",
        "\n",
        "from transformers import TextStreamer\n",
        "text_streamer = TextStreamer(tokenizer, skip_prompt = True)\n",
        "_ = model.generate(input_ids = inputs, streamer = text_streamer,\n",
        "                   max_new_tokens = 256, temperature = 0.1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f422JgM9sdVT"
      },
      "source": [
        "### Saving to float16 for VLLM\n",
        "\n",
        "We also support saving to `float16` directly. Select `merged_16bit` for float16 or `merged_4bit` for int4. We also allow `lora` adapters as a fallback. Use `push_to_hub_merged` to upload to your Hugging Face account! You can go to https://huggingface.co/settings/tokens for your personal tokens."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "iHjt_SMYsd3P"
      },
      "outputs": [],
      "source": [
        "# Merge to 16bit\n",
        "if False:\n",
        "    model.save_pretrained_merged(\"model\", tokenizer, save_method = \"merged_16bit\",)\n",
        "if False: # Change to True to upload finetune\n",
        "    model.push_to_hub_merged(\"hf/model\", tokenizer, save_method = \"merged_16bit\", token = \"\")\n",
        "\n",
        "# Merge to 4bit\n",
        "if False:\n",
        "    model.save_pretrained_merged(\"model\", tokenizer, save_method = \"merged_4bit\",)\n",
        "if False: # Change to True to upload finetune\n",
        "    model.push_to_hub_merged(\"hf/model\", tokenizer, save_method = \"merged_4bit\", token = \"\")\n",
        "\n",
        "# Just LoRA adapters\n",
        "if False:\n",
        "    model.save_pretrained_merged(\"model\", tokenizer, save_method = \"lora\",)\n",
        "if False: # Change to True to upload finetune\n",
        "    model.push_to_hub_merged(\"hf/model\", tokenizer, save_method = \"lora\", token = \"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TCv4vXHd61i7"
      },
      "source": [
        "### GGUF / llama.cpp Conversion\n",
        "To save to `GGUF` / `llama.cpp`, we support it natively now! We clone `llama.cpp` and we default save it to `q8_0`. We allow all methods like `q4_k_m`. Use `save_pretrained_gguf` for local saving and `push_to_hub_gguf` for uploading to HF.\n",
        "\n",
        "Some supported quant methods (full list on our [Wiki page](https://github.com/unslothai/unsloth/wiki#gguf-quantization-options)):\n",
        "* `q8_0` - Fast conversion. High resource use, but generally acceptable.\n",
        "* `q4_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q4_K.\n",
        "* `q5_k_m` - Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q5_K."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FqfebeAdT073"
      },
      "outputs": [],
      "source": [
        "# Save to 8bit Q8_0\n",
        "if False:\n",
        "    model.save_pretrained_gguf(\"model\", tokenizer,)\n",
        "if False: # Change to True to upload finetune\n",
        "    model.push_to_hub_gguf(\"hf/model\", tokenizer, token = \"\")\n",
        "\n",
        "# Save to 16bit GGUF\n",
        "if False:\n",
        "    model.save_pretrained_gguf(\"model\", tokenizer, quantization_method = \"f16\")\n",
        "if False: # Change to True to upload finetune\n",
        "    model.push_to_hub_gguf(\"hf/model\", tokenizer, quantization_method = \"f16\", token = \"\")\n",
        "\n",
        "# Save to q4_k_m GGUF\n",
        "if False:\n",
        "    model.save_pretrained_gguf(\"model\", tokenizer, quantization_method = \"q4_k_m\")\n",
        "if False: # Change to True to upload finetune\n",
        "    model.push_to_hub_gguf(\"hf/model\", tokenizer, quantization_method = \"q4_k_m\", token = \"\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UyxqD2WJvD5n"
      },
      "source": [
        "Now, use the `model-unsloth.gguf` file or `model-unsloth-Q4_K_M.gguf` file in llama.cpp or a UI based system like Jan or Open WebUI. You can install Jan [here](https://github.com/janhq/jan) and Open WebUI [here](https://github.com/open-webui/open-webui)\n",
        "\n",
        "And we're done! If you have any questions on Unsloth, we have a [Discord](https://discord.gg/unsloth) channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!\n",
        "\n",
        "Some other links:\n",
        "1. Train your own reasoning model - Llama GRPO notebook [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.1_(8B)-GRPO.ipynb)\n",
        "2. Saving finetunes to Ollama. [Free notebook](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3_(8B)-Ollama.ipynb)\n",
        "3. Llama 3.2 Vision finetuning - Radiography use case. [Free Colab](https://colab.research.google.com/github/unslothai/notebooks/blob/main/nb/Llama3.2_(11B)-Vision.ipynb)\n",
        "6. See notebooks for DPO, ORPO, Continued pretraining, conversational finetuning and more on our [documentation](https://docs.unsloth.ai/get-started/unsloth-notebooks)!\n",
        "\n",
        "<div class=\"align-center\">\n",
        "  <a href=\"https://unsloth.ai\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/unsloth%20new%20logo.png\" width=\"115\"></a>\n",
        "  <a href=\"https://discord.gg/unsloth\"><img src=\"https://github.com/unslothai/unsloth/raw/main/images/Discord.png\" width=\"145\"></a>\n",
        "  <a href=\"https://docs.unsloth.ai/\"><img src=\"https://github.com/unslothai/unsloth/blob/main/images/documentation%20green%20button.png?raw=true\" width=\"125\"></a>\n",
        "\n",
        "  Join Discord if you need help + ⭐️ <i>Star us on <a href=\"https://github.com/unslothai/unsloth\">Github</a> </i> ⭐️\n",
        "</div>\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "Qb5ekq58-Uig"
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "gpuType": "T4",
      "provenance": []
    },
    "kernelspec": {
      "display_name": ".venv",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.16"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
